Interactive Visual Music: A Personal Perspective

25 Interactive performance is one of the most innovative ways computers can be used in music, and it leads to new ways of thinking about music composition and performance. Interactive performance also poses many technical challenges, resulting in new languages and special hardware including sensors, synthesis methods, and software techniques. As if there are not already enough problems to tackle, many composers and artists have explored the combination of computer animation and music within interactive performances. In this article, I describe my own work in this area, dating from around 1987, including discussions of artistic and technical challenges as they have evolved. I also describe the Aura system, which I now use to create interactive music, animation, and video. My main interest in composing and developing systems is to approach music and video as expressions of the same underlying musical “deep structure.” Thus, images are not an interpretation or accompaniment to audio but rather an integral part of the music and the listener/viewer experience. From a systems perspective, I have always felt that timing and responsiveness are critical if images and music are to work together. I have focused on software organizations that afford precise timing and flexible interaction, sometimes at the expense of rich imagery or more convenient tools for imagemaking. “Interaction” is commonly accepted (usually without much thought) as a desirable property of computer systems, so the term has become overused and vague. A bit of explanation is in order. Interaction is a two-way flow of information between live performer(s) and computer. In my work, I see improvisation as a way to use the talents of the performer to the fullest. Once a performer is given the freedom to improvise, however, it is difficult for a composer to retain much control. I resolve this problem by composing interaction. In other words, rather than writing notes for the performer, I design musical situations by generating sounds and images that encourage improvising performers to make certain musical choices. In this way, I feel that I can achieve compositional control over both large-scale structure and fine-grain musical texture. At the same time, performers are empowered to augment the composed interactive framework with their own ideas, musical personality, and sounds.